{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fefeca4e-2808-4f33-a346-07f7e12bc59e",
   "metadata": {},
   "source": [
    "# OpenVINO Tokenizers: Incorporate Text Processing Into OpenVINO Pipelines\n",
    "\n",
    "<center><img src=\"https://github.com/openvinotoolkit/openvino_notebooks/assets/51917466/047f9167-a4ef-4d3d-a33b-d124541f9e2c\"></center>\n",
    "\n",
    "OpenVINO Tokenizers is an OpenVINO extension and a Python library designed to streamline tokenizer conversion for seamless integration into your projects. It supports Python and C++ environments and is compatible with all major platforms: Linux, Windows, and MacOS.\n",
    "\n",
    "\n",
    "#### Table of contents:\n",
    "- [Tokenization Basics](#Tokenization-Basics)\n",
    "- [Acquiring OpenVINO Tokenizers](#Acquiring-OpenVINO-Tokenizers)\n",
    "    - [Convert Tokenizer from HuggingFace Hub with CLI Tool](#Convert-Tokenizer-from_HuggingFace-Hub-with-CLI-Tool)\n",
    "    - [Convert Tokenizer from HuggingFace Hub with Python API](#Convert-Tokenizer-from-HuggingFace-Hub-with-Python-API)\n",
    "- [Text Generation Pipeline with OpenVINO Tokenizers](#Text-Generation-Pipeline-with-OpenVINO-Tokenizers)\n",
    "- [Text Generation Pipeline with OpenVINO GenAI and OpenVINO Tokenizers](#text-generation-pipeline-with-openvino-genai-and-openvino-tokenizers)\n",
    "- [Merge Tokenizer into a Model](#Merge-Tokenizer-into-a-Model)\n",
    "- [Conclusion](#Conclusion)\n",
    "- [Links](#Links)\n",
    "\n",
    "\n",
    "## Tokenization Basics\n",
    "\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "One does not simply put text into a neural network, only numbers. The process of transforming text into a sequence of numbers is called **tokenization**. It usually contains several steps that transform the original string, splitting it into parts - tokens - with an associated number in a dictionary. You can check the [interactive GPT-4 tokenizer](https://platform.openai.com/tokenizer) to gain an intuitive understanding of the principles of tokenizer work.\n",
    "\n",
    "<center><img src=\"https://github.com/openvinotoolkit/openvino_notebooks/assets/51917466/55cc2420-da2a-4022-bae3-84d566a68e4c\"></center>\n",
    "\n",
    "There are two important points in the tokenizer-model relation:\n",
    "1. Every neural network with text input is paired with a tokenizer and _cannot be used without it_.\n",
    "2. To reproduce the model's accuracy on a specific task, it is essential to _utilize the same tokenizer employed during the model training_.\n",
    "\n",
    "That is why almost all model repositories on [HuggingFace Hub](https://HuggingFace.co/models) also contain tokenizer files (`tokenizer.json`, `vocab.txt`, `merges.txt`, etc.).\n",
    "\n",
    "The process of transforming a sequence of numbers into a string is called **detokenization**. Detokenizer can share the token dictionary with a tokenizer, like any LLM chat model, or operate with an entirely distinct dictionary. For instance, translation models dealing with different source and target languages often necessitate separate dictionaries.\n",
    "\n",
    "<center><img src=\"https://github.com/openvinotoolkit/openvino_notebooks/assets/51917466/942f1692-b7f2-4aca-93ef-32e9866c204f\"></center>\n",
    "\n",
    "Some tasks only need a tokenizer, like text classification, named entity recognition, question answering, and feature extraction. On the other hand, for tasks such as text generation, chat, translation, and abstractive summarization, both a tokenizer and a detokenizer are required.\n",
    "### Installation Instructions\n",
    "\n",
    "This is a self-contained example that relies solely on its own code.\n",
    "\n",
    "We recommend  running the notebook in a virtual environment. You only need a Jupyter server to start.\n",
    "For details, please refer to [Installation Guide](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/README.md#-installation-guide).\n",
    "\n",
    "<img referrerpolicy=\"no-referrer-when-downgrade\" src=\"https://static.scarf.sh/a.png?x-pxid=5b5a4db0-7875-4bfb-bdbd-01698b5b1a77&file=notebooks/openvino-tokenizers/openvino-tokenizers.ipynb\" />\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8d502090-78eb-454c-a7a6-03b128f95ff6",
   "metadata": {},
   "source": [
    "## Acquiring OpenVINO Tokenizers\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "OpenVINO Tokenizers Python library allows you to convert HuggingFace tokenizers into OpenVINO models. To install all required dependencies use `pip install openvino-tokenizers[transformers]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c698c94-b852-4b06-b699-7d417fb55e10",
   "metadata": {},
   "outputs": [],
   "source": [
    "import platform\n",
    "\n",
    "%pip uninstall -q -y optimum optimum-intel optimum-onnx\n",
    "%pip install -q -U \"openvino>=2024.3.0\" openvino-tokenizers[transformers] openvino-genai accelerate --extra-index-url https://download.pytorch.org/whl/cpu\n",
    "%pip install \"torch==2.8\" \"torchvision==0.23.0\" --extra-index-url https://download.pytorch.org/whl/cpu\n",
    "\n",
    "if platform.system() == \"Darwin\":\n",
    "    %pip install \"numpy<2.0.0\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95899591",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from pathlib import Path\n",
    "\n",
    "if not Path(\"notebook_utils.py\").exists():\n",
    "    r = requests.get(\n",
    "        url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py\",\n",
    "    )\n",
    "    open(\"notebook_utils.py\", \"w\").write(r.text)\n",
    "\n",
    "if not Path(\"cmd_helper.py\").exists():\n",
    "    r = requests.get(\n",
    "        url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/cmd_helper.py\",\n",
    "    )\n",
    "    open(\"cmd_helper.py\", \"w\").write(r.text)\n",
    "\n",
    "# Read more about telemetry collection at https://github.com/openvinotoolkit/openvino_notebooks?tab=readme-ov-file#-telemetry\n",
    "from notebook_utils import collect_telemetry\n",
    "\n",
    "collect_telemetry(\"openvino-tokenizers.ipynb\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "576b3d35-1282-4328-aa11-e871759a6dbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "\n",
    "tokenizer_dir = Path(\"tokenizer/\")\n",
    "model_id = \"TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0da69135-265e-482f-a399-521e3980dec7",
   "metadata": {},
   "source": [
    "### Convert Tokenizer from HuggingFace Hub with CLI Tool\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "The first way is to use the CLI utility, bundled with OpenVINO Tokenizers. Use `--with-detokenizer` flag to add the detokenizer model to the output. By setting `--clean-up-tokenization-spaces=False` we ensure that the detokenizer correctly decodes a code-generation model output. `--trust-remote-code` flag works the same way as passing `trust_remote_code=True` to `AutoTokenizer.from_pretrained` constructor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f8ff589d-2f44-4812-894a-ba25a826292e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading Huggingface Tokenizer...\n",
      "Converting Huggingface Tokenizer to OpenVINO...\n",
      "Saved OpenVINO Tokenizer: tokenizer/openvino_tokenizer.xml, tokenizer/openvino_tokenizer.bin\n",
      "Saved OpenVINO Detokenizer: tokenizer/openvino_detokenizer.xml, tokenizer/openvino_detokenizer.bin\n"
     ]
    }
   ],
   "source": [
    "if not tokenizer_dir.exists():\n",
    "    !convert_tokenizer $model_id --with-detokenizer -o $tokenizer_dir"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e6cbb08b-f980-458e-8c5b-0d752d47aa26",
   "metadata": {},
   "source": [
    "> ⚠️ If you have any problems with the command above on MacOS, try to [install tbb](https://formulae.brew.sh/formula/tbb#default).\n",
    "\n",
    "The result is two OpenVINO models: `openvino_tokenizer` and `openvino_detokenizer`. Both can be interacted with using `read_model`, `compile_model` and `save_model`, similar to any other OpenVINO model.\n",
    "\n",
    "### Convert Tokenizer from HuggingFace Hub with Python API\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "The other method is to pass HuggingFace `hf_tokenizer` object to `convert_tokenizer` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a517c20c-b375-4ebc-b8a5-9eaae31bf62b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<Model: 'tokenizer'\n",
       " inputs[\n",
       " <ConstOutput: names[string_input] shape[?] type: string>\n",
       " ]\n",
       " outputs[\n",
       " <ConstOutput: names[input_ids] shape[?,?] type: i64>,\n",
       " <ConstOutput: names[attention_mask] shape[?,?] type: i64>\n",
       " ]>,\n",
       " <Model: 'detokenizer'\n",
       " inputs[\n",
       " <ConstOutput: names[Parameter_22] shape[?,?] type: i64>\n",
       " ]\n",
       " outputs[\n",
       " <ConstOutput: names[string_output] shape[?] type: string>\n",
       " ]>)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import AutoTokenizer\n",
    "from openvino_tokenizers import convert_tokenizer\n",
    "\n",
    "\n",
    "hf_tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
    "ov_tokenizer, ov_detokenizer = convert_tokenizer(hf_tokenizer, with_detokenizer=True)\n",
    "ov_tokenizer, ov_detokenizer"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cd211b9f-37a9-4ae7-bc3e-4619643c08b8",
   "metadata": {},
   "source": [
    "That way you get OpenVINO model objects. Use `save_model` function from OpenVINO to reuse converted tokenizers later:\n",
    "\n",
    "> ⚠️ Import `openvino_tokenizers` will add all tokenizer-related operations to OpenVINO, after which you can work with saved tokenizers and detokenizers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "eefda978-2668-4486-825e-30a1afbd240b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openvino as ov\n",
    "\n",
    "# This import is needed to add all tokenizer-related operations to OpenVINO\n",
    "import openvino_tokenizers  # noqa: F401\n",
    "\n",
    "\n",
    "ov.save_model(ov_tokenizer, tokenizer_dir / \"openvino_tokenizer.xml\")\n",
    "ov.save_model(ov_detokenizer, tokenizer_dir / \"openvino_detokenizer.xml\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0b16883a-2a69-4ae7-ab57-99d9b35784e1",
   "metadata": {},
   "source": [
    "To use the tokenizer, compile the converted model and input a list of strings.  It's essential to be aware that not all original tokenizers support multiple strings (also called batches) as input.  This limitation arises from the requirement for all resulting number sequences to maintain the same length. To address this, a padding token must be specified, which will be appended to shorter tokenized strings. In cases where no padding token is determined in the original tokenizer, OpenVINO Tokenizers defaults to using $0$ for padding.  Presently, _only right-side padding is supported_, typically used for classification tasks, but not suitable for text generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0fb96120-57f4-4f2d-b585-84ce0367cb51",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token ids: [[   1 4321]\n",
      " [   1 6031]]\n",
      "Detokenized text: ['Test' 'strings']\n"
     ]
    }
   ],
   "source": [
    "tokenizer, detokenizer = ov.compile_model(ov_tokenizer), ov.compile_model(ov_detokenizer)\n",
    "test_strings = [\"Test\", \"strings\"]\n",
    "\n",
    "token_ids = tokenizer(test_strings)[\"input_ids\"]\n",
    "print(f\"Token ids: {token_ids}\")\n",
    "\n",
    "detokenized_text = detokenizer(token_ids)[\"string_output\"]\n",
    "print(f\"Detokenized text: {detokenized_text}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e602d7b3-b1cb-46eb-9738-4fbb41bb2b83",
   "metadata": {},
   "source": [
    "We can compare the result of converted (de)tokenizer with the original one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "753727b4-0318-4806-90f3-a9c9c56a4297",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token ids: [[1, 4321], [1, 6031]]\n",
      "Detokenized text: ['<s> Test', '<s> strings']\n"
     ]
    }
   ],
   "source": [
    "hf_token_ids = hf_tokenizer(test_strings).input_ids\n",
    "print(f\"Token ids: {hf_token_ids}\")\n",
    "\n",
    "hf_detokenized_text = hf_tokenizer.batch_decode(hf_token_ids)\n",
    "print(f\"Detokenized text: {hf_detokenized_text}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9deb9c29-31e0-436c-9f52-4e396a37283f",
   "metadata": {},
   "source": [
    "## Text Generation Pipeline with OpenVINO Tokenizers\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Let's build a text generation pipeline with OpenVINO Tokenizers and minimal dependencies. To obtain an OpenVINO model we will use the Optimum library. The latest version allows you to get a so-called [stateful model](https://docs.openvino.ai/2024/openvino-workflow/running-inference/stateful-models.html).\n",
    "\n",
    "The original `TinyLlama-1.1B-intermediate-step-1431k-3T` model is 4.4Gb. To reduce network and disk usage we will load a converted model which has also been compressed to `int8`. The original conversion command is commented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a9870da3-477f-49c5-af80-ef081f92abeb",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_dir = Path(Path(model_id).name)\n",
    "\n",
    "if not model_dir.exists():\n",
    "    # Converting the original model\n",
    "    # %pip install -U \"git+https://github.com/huggingface/optimum-intel.git\" \"nncf>=2.8.0\" onnx\n",
    "    # %optimum-cli export openvino -m $model_id --task text-generation-with-past $model_dir\n",
    "\n",
    "    # Load already converted model\n",
    "    from huggingface_hub import hf_hub_download\n",
    "\n",
    "    hf_hub_download(\n",
    "        \"chgk13/TinyLlama-1.1B-intermediate-step-1431k-3T\",\n",
    "        filename=\"openvino_model.xml\",\n",
    "        local_dir=model_dir,\n",
    "    )\n",
    "    hf_hub_download(\n",
    "        \"chgk13/TinyLlama-1.1B-intermediate-step-1431k-3T\",\n",
    "        filename=\"openvino_model.bin\",\n",
    "        local_dir=model_dir,\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "fe7c0342-0e40-4f17-8340-a0ff0c99b6f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from tqdm.notebook import trange\n",
    "from pathlib import Path\n",
    "from openvino_tokenizers.constants import EOS_TOKEN_ID_NAME\n",
    "\n",
    "\n",
    "core = ov.Core()\n",
    "\n",
    "ov_model = core.read_model(model_dir / \"openvino_model.xml\")\n",
    "compiled_model = core.compile_model(ov_model)\n",
    "infer_request = compiled_model.create_infer_request()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f6f53a35-446d-4a07-9e67-5794a53b12ba",
   "metadata": {},
   "source": [
    "The `infer_request` object provides control over the model's state - a Key-Value cache that speeds up inference by reducing computations. Multiple inference requests can be created, and each request maintains a distinct and separate state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "58a493cc-e29d-46b5-bb6c-d28e17651d85",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9135a4ff0a7141949c4990d803862bc0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt:\n",
      "Quick brown fox jumped\n",
      "Generated:\n",
      "over the fence.\n"
     ]
    }
   ],
   "source": [
    "text_input = [\"Quick brown fox jumped\"]\n",
    "\n",
    "model_input = {name.any_name: output for name, output in tokenizer(text_input).items()}\n",
    "\n",
    "if \"position_ids\" in (input.any_name for input in infer_request.model_inputs):\n",
    "    model_input[\"position_ids\"] = np.arange(model_input[\"input_ids\"].shape[1], dtype=np.int64)[np.newaxis, :]\n",
    "\n",
    "# No beam search, set idx to 0\n",
    "model_input[\"beam_idx\"] = np.array([0], dtype=np.int32)\n",
    "\n",
    "# End of sentence token is that model signifies the end of text generation\n",
    "# Read EOS token ID from rt_info of tokenizer/detokenizer ov.Model object\n",
    "eos_token = ov_tokenizer.get_rt_info(EOS_TOKEN_ID_NAME).value\n",
    "\n",
    "tokens_result = np.array([[]], dtype=np.int64)\n",
    "\n",
    "# Reset KV cache inside the model before inference\n",
    "infer_request.reset_state()\n",
    "max_infer = 5\n",
    "\n",
    "for _ in trange(max_infer):\n",
    "    infer_request.start_async(model_input)\n",
    "    infer_request.wait()\n",
    "\n",
    "    output_tensor = infer_request.get_output_tensor()\n",
    "\n",
    "    # Get the most probable token\n",
    "    token_indices = np.argmax(output_tensor.data, axis=-1)\n",
    "    output_token = token_indices[:, -1:]\n",
    "\n",
    "    # Concatenate previous tokens result with newly generated token\n",
    "    tokens_result = np.hstack((tokens_result, output_token))\n",
    "    if output_token[0, 0] == eos_token:\n",
    "        break\n",
    "\n",
    "    # Prepare input for the next inference iteration\n",
    "    model_input[\"input_ids\"] = output_token\n",
    "    model_input[\"attention_mask\"] = np.hstack((model_input[\"attention_mask\"].data, [[1]]))\n",
    "    model_input[\"position_ids\"] = np.hstack(\n",
    "        (\n",
    "            model_input[\"position_ids\"].data,\n",
    "            [[model_input[\"position_ids\"].data.shape[-1]]],\n",
    "        )\n",
    "    )\n",
    "\n",
    "\n",
    "text_result = detokenizer(tokens_result)[\"string_output\"]\n",
    "print(f\"Prompt:\\n{text_input[0]}\")\n",
    "print(f\"Generated:\\n{text_result[0]}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "691e7415",
   "metadata": {},
   "source": [
    "## Text Generation Pipeline with OpenVINO GenAI and OpenVINO Tokenizers\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "[OpenVINO GenAI](https://github.com/openvinotoolkit/openvino.genai) is a flavor of OpenVINO, aiming to simplify running inference of generative AI models. It hides the complexity of the generation process and minimizes the amount of code required.\n",
    "OpenVINO GenAI depends on [OpenVINO](https://github.com/openvinotoolkit/openvino) and [OpenVINO Tokenizers](https://github.com/openvinotoolkit/openvino_tokenizers).\n",
    "\n",
    "Firstly we need to create a pipeline with `LLMPipeline`. `LLMPipeline` is the main object used for text generation using LLM in OpenVINO GenAI API. You can construct it straight away from the folder where both converted model and tokenizer are located, e.g. `ov_genai.LLMPipeline(model_and_tokenizer_path)`. \n",
    "\n",
    "As the model and tokenizer are located in different directories, we create a `ov_genai.Tokenizer` object by providing the path to saved tokenizer. Then we will provide directory with model, tokenizer object and device for `LLMPipeline`. Lastly we run `generate` method and get the output in text format.\n",
    "\n",
    "Additionally, we can configure parameters for decoding. We can get the default config with `get_generation_config()`, setup parameters, and apply the updated version with `set_generation_config(config)` or put config directly to `generate()`. It's also possible to specify the needed options just as inputs in the `generate()` method, as shown below, e.g. we can add `max_new_tokens` to stop generation if a specified number of tokens is generated and the end of generation is not reached.\n",
    "\n",
    "Let's build the same text generation pipeline, but with simplified Python [OpenVINO Generate API](https://github.com/openvinotoolkit/openvino.genai/blob/master/src/README.md). We will use the same model and tokenizer downloaded in previous steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "ce5fb0bf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt:\n",
      "Quick brown fox jumped\n",
      "Generated:\n",
      "over the lazy dog.\n"
     ]
    }
   ],
   "source": [
    "import openvino_genai as ov_genai\n",
    "\n",
    "genai_tokenizer = ov_genai.Tokenizer(str(tokenizer_dir))\n",
    "pipe = ov_genai.LLMPipeline(str(model_dir), genai_tokenizer, \"CPU\")\n",
    "\n",
    "result = pipe.generate(text_input[0], max_new_tokens=max_infer)\n",
    "\n",
    "print(f\"Prompt:\\n{text_input[0]}\")\n",
    "print(f\"Generated:\\n{result}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7beb6ec6-2484-44ce-b61b-c7ae605dffee",
   "metadata": {},
   "source": [
    "## Merge Tokenizer into a Model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Packages like `tensorflow-text` offer the convenience of integrating text processing directly into the model, streamlining both distribution and usage.  Similarly, with OpenVINO Tokenizers, you can create models that combine a converted tokenizer and a model. It's important to note that not all scenarios benefit from this merge. In cases where a tokenizer is used once and a model is inferred multiple times, as seen in the earlier text generation example, maintaining a separate (de)tokenizer and model is advisable to prevent unnecessary tokenization-detokenization cycles during inference. Conversely, if both a tokenizer and a model are used once in each pipeline inference, merging simplifies the workflow and aids in avoiding the creation of intermediate objects:\n",
    "\n",
    "<center><img src=\"https://github.com/openvinotoolkit/openvino_notebooks/assets/51917466/d4ece285-e445-4b76-a1ab-356427900860\"></center>\n",
    "\n",
    "The OpenVINO Python API allows you to avoid this by using the `share_inputs` option during inference, but it requires additional input from a developer every time the model is inferred. Combining the models and tokenizers simplifies memory management.\n",
    "Moreover, after the combining models inputs have changed - original model has three inputs (`input_ids`, `attention_mask`, `token_type_ids`) and combined model has only one input for text input prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "c044b56b-dae0-4fdb-97df-2aa555285f35",
   "metadata": {
    "test_replace": {
     "optimum_cli(model_id, model_dir, additional_args={\"task\": \"text-classification\"})": "optimum_cli(model_id, model_dir, additional_args={\"task\": \"text-classification\"}, debug_logs=True)"
    }
   },
   "outputs": [],
   "source": [
    "from cmd_helper import optimum_cli\n",
    "\n",
    "model_id = \"mrm8488/bert-tiny-finetuned-sms-spam-detection\"\n",
    "model_dir = Path(Path(model_id).name)\n",
    "\n",
    "if not model_dir.exists():\n",
    "    %pip install -qU \"git+https://github.com/huggingface/optimum-intel.git\" --extra-index-url https://download.pytorch.org/whl/cpu\n",
    "    optimum_cli(model_id, model_dir, additional_args={\"task\": \"text-classification\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "c6a42d4c-1982-41b9-9612-aa19138518ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original OpenVINO model inputs:\n",
      "<Output: names[input_ids] shape[?,?] type: i64>\n",
      "<Output: names[attention_mask] shape[?,?] type: i64>\n",
      "<Output: names[token_type_ids] shape[?,?] type: i64>\n",
      "\n",
      "Combined OpenVINO model inputs:\n",
      "<Output: names[Parameter_4430] shape[?] type: string>\n",
      "\n",
      "Logits: [[ 1.2007061 -1.469803 ]]\n"
     ]
    }
   ],
   "source": [
    "from openvino_tokenizers import connect_models\n",
    "\n",
    "\n",
    "core = ov.Core()\n",
    "text_input = [\"Free money!!!\"]\n",
    "\n",
    "ov_tokenizer = core.read_model(model_dir / \"openvino_tokenizer.xml\")\n",
    "ov_model = core.read_model(model_dir / \"openvino_model.xml\")\n",
    "combined_model = connect_models(ov_tokenizer, ov_model)\n",
    "ov.save_model(combined_model, model_dir / \"combined_openvino_model.xml\")\n",
    "\n",
    "print(\"Original OpenVINO model inputs:\")\n",
    "for input in ov_model.inputs:\n",
    "    print(input)\n",
    "\n",
    "print(\"\\nCombined OpenVINO model inputs:\")\n",
    "for input in combined_model.inputs:\n",
    "    print(input)\n",
    "\n",
    "compiled_combined_model = core.compile_model(combined_model)\n",
    "openvino_output = compiled_combined_model(text_input)\n",
    "\n",
    "print(f\"\\nLogits: {openvino_output['logits']}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5e5e3084-e206-4b39-8ca0-2ee087af10ef",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "The OpenVINO Tokenizers integrate text processing operations into the OpenVINO ecosystem. Enabling the conversion of HuggingFace tokenizers into OpenVINO models, the library allows efficient deployment of deep learning pipelines across varied environments. The feature of combining tokenizers and models not only simplifies memory management but also helps to streamline model usage and deployment.\n",
    "\n",
    "\n",
    "## Links\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "- [Installation instructions for different environments](https://github.com/openvinotoolkit/openvino_tokenizers?tab=readme-ov-file#installation)\n",
    "- [Supported Tokenizer Types](https://github.com/openvinotoolkit/openvino_tokenizers?tab=readme-ov-file#supported-tokenizer-types)\n",
    "- [OpenVINO.GenAI repository with the C++ example of OpenVINO Tokenizers usage](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/cpp/greedy_causal_lm)\n",
    "- [HuggingFace Tokenizers Comparison Table](https://github.com/openvinotoolkit/openvino_tokenizers?tab=readme-ov-file#output-match-by-model)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  },
  "openvino_notebooks": {
   "imageUrl": "https://github.com/openvinotoolkit/openvino_notebooks/assets/51917466/047f9167-a4ef-4d3d-a33b-d124541f9e2c",
   "tags": {
    "categories": [
     "API Overview",
     "Convert",
     "Optimize"
    ],
    "libraries": [],
    "other": [],
    "tasks": [
     "Text Classification",
     "Text Generation"
    ]
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
